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TITLE OF THE INVENTION 

THREE DIMENSIONAL VOLUMETRIC DISPLAY INPUT AND OUTPUT CONFIGURATIONS 
CROSS-REFERENCE TO RELATED APPLICATION(S) 

[0001] This application is related to, is a continuation of and claims priority to U.S. application 
entitled Three Dimensional Volumetric Display Input and Output Configurations, having serial 
number 10/183,970, by Kurtenbach et al., filed June 28, 2002 (allowed), which claims priority to 
U.S. provisional application entitled User Interfaces For Volumetric Displays, having serial 
number 60/350,952 (S&H Docket 1252.1054P), by Kurtenbach et al, filed January 25, 2002. 
This application is also related to U.S. application entitled Volume Management System For 
Volumetric Displays, having serial number 10/183,966 (S&H Docket 1252.1065), by Kurtenbach 
et al, filed June 28, 2002, to U.S. application entitled Widgets Displayed And Operable On A 
Surface Of A Volumetric Display Enclosure, having serial number 10/183,945 (S&H Docket 
1252.1066) by Fitzmaurice et al, filed June 28, 2002, to U.S. application entitled Graphical User 
Interface Widgets Viewable And Readable From Multiple Viewpoints In A Volumetric Display, 
having serial number 10/183,968 (S&H Docket 1252.1067), by Fitzmaurice et al, filed June 28, 
2002, to U.S. application entitled A System For Physical Rotation of Volumetric Display 
Enclosures To Facilitate Viewing, having serial number 10/183,765 (S&H Docket 1252.1068), by 
Balakrishnan et al, filed June 28, 2002, and to U.S. application entitled Techniques For Pointing 
To Locations Within A Volumetric Display, having serial number 10/183,944 (S&H Docket 
1252.1069), by Balakrishnan et al, filed June 28, 2002, and all of which are incorporated by 
reference herein. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0002] The present invention is directed to input and output configurations for three- 
dimensional volumetric displays and, more particularly, to input configurations that allow the 
content of a three-dimensional volumetric display output configuration to be affected by actions 
by a user operating within an input configuration. 

2. Description of the Related Art 

[0003] A class of three-dimensional (3D) displays, called volumetric displays, is currently 
undergoing rapid advancement. The types of displays in this class include holographic displays, 
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swept volume displays and static volume displays. Volumetric displays allow for three- 
dimensional (3D) graphical scenes to be displayed within a true 3D volume. Such displays can 
take many shapes including cylinders, globes, domes, cubes, an arbitrary shape, etc. with a 
dome being a typical shape. Because the technology of these displays is undergoing rapid 
development those of skill in the art are concentrating on the engineering of the display itself. 
As a result, the man-machine interface to or input/output configurations with which people 
interface with these types of displays is receiving scant attention. 

[0004] While the volumetric displays allow a user to view different parts of a true 3D scene, 
the act of viewing the different parts typically requires that the user physically move around (or 
over) the display or that the display be moved or rotated in front of the user. As the display 
moves relative to the user, graphical objects may also move relative to the user. When the 
display is relatively stationary or when it is relatively moving, the user may need to interact with 
the display. As a result, what the user needs is an effective mechanism for interacting with the 
display. 

SUMMARY OF THE INVENTION 

[0005] It is an aspect of the present invention to provide effective mechanisms for a user to 
interact with content of the three-dimensional volumetric display. 

[0006] It is also an aspect of the present invention to provide input and output configurations 
for a three-dimensional volumetric display. 

[0007] It is another aspect of the present invention to provide dome, cubical and cylindrical 
output configurations. 

[0008] It is also an aspect of the present invention to provide input configurations that allow a 
3D volumetric input space to be mapped to the 3D volumetric display, a planer 2D input space 
to be mapped to the 3D volumetric display, a planar 2D input space to be mapped to a planar 
2D space within the 3D volumetric display, and a non-planar 2D input space to be mapped to 
the 3D volumetric display. 

[0009] The above aspects can be attained by a system that allows a number of 3D 
volumetric display configurations, such as dome, cubical and cylindrical volumetric display 
enclosures, to interact with a number of different input configurations, for example, a three- 
dimensional position sensing system, a planar position sensing system and a non-planar 
position sensing system. The user interacts with the input configurations, such as by moving a 
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stylus on a sensing grid formed on an enclosure surface. This interaction affects the content of 
the volumetric display, for example, by moving a cursor within the 3D display space of the 
volumetric display. 

[0010] These together with other aspects and advantages which will be subsequently 
apparent, reside in the details of construction and operation as more fully hereinafter described 
and claimed, reference being had to the accompanying drawings forming a part hereof, wherein 
like numerals refer to like parts throughout. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] Figure 1 shows a volumetric display. 

[0012] Figures 2, 3 and 4 depict a 3D to 3D system configurations. 

[0013] Figures 5, 6 and 7 depict 2D to 3D configurations. 

[0014] Figure 8 shows a non-planar to 3D configuration. 

[0015] Figures 9, 10, 1 1 , 12, 13 and 14 show configurations with physical intermediaries. 

[001 6] Figure 1 5 depicts components of the system 

[0017] Figures 16A, 16B, 16C and 16D illustrate digitizer embodiments. 

[0018] Figures 17 illustrates components of a digitizer. 

[0019] Figures 18A, 18B and 18C show details of a dome shaped digitizer. 

[0020] Figure 1 9 depicts the operations of the system. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0021] Volumetric displays allow a user to have a true three-dimensional (3D) view of a 
scene 12 and are typically provided in the form of a dome 14, as depicted in figure 1 . The user 
16, as can be surmised from figure 1 , can move about the dome 14 to view different parts of the 
scene 12. From a particular arbitrary viewpoint or position, a user may want to interact with the 
scene or content within the volumetric display. 

[0022] There are a number of different solutions to this problem. These solutions involve 
creating input/output configurations for the volumetric display that define a spatial 
correspondence between an input space and an output space. The configurations also define a 
dynamically updatable spatial correspondence of the input and output spaces with the user. 
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[0023] In a first solution, a 3D volumetric input space is mapped to a 3D volumetric display 
space. In one configuration, as depicted in figure 2, the user's hand 30 is tracked via a glove or 
a set of cameras in a volume 32 directly below the display volume 34. A virtual representation 
of the hand 36, or some other type of position indicator, such as a cursor, is superimposed into 
the 3D output volumetric display 34. In a second configuration, as depicted in figure 3, the 3D 
display 50 is surrounded by a 3D input space 52, created by a 3D volume input system, such as 
the Flock of Birds system from Ascension Technology Corporation. In this configuration, the 
user's hand 54, including a position indicator/sensor, is mapped to a cursor 56 or some other 
position indicator representation, such as a virtual hand, within the display 50. The position 
sensor also produces a vector that indicates which direction the sensor is pointing. The vector 
can be used to create a cursor in the enclosure at a fixed position along the vector. Rather than 
using the vector produced by the position sensor, the system infers an input vector based on the 
position of the input device and the center of the display. This spatial relationship or 
correspondence between the input space, output space and user position is dynamically 
updated as the user moves about the display. That is, the input/output space is automatically 
compensated/reconfigured. Another configuration is to use half-silvered mirrors 70 (see figure 
4) to combine the volumetric image 72 with the user's view of their hands in a hand movement 
volume. This way, the user sees their hands operating within the display. Another alternative is 
to use a camera to capture the users hands in the input space and superimpose them onto the 
volumetric display space. Another alternative is an augmented-reality system where the user 
has a see-through, head mounted display (2D) which is being tracked. As the user moves the 
position and orientation of their head, graphics are presented on the LCD display and are 
aligned with real-world objects. 

[0024] Another solution is to map a planer 2D input space into a 3D output space. This is 
particularly useful in controlling some subset of the 3D volumetric display space. For example, 
a standard 2D digitizing tablet or digitizer 90 (see figure 5) or a regular mouse can be mapped 
to control aspects of the 3D scene, such as moving 3D objects along two dimensions. 

[0025] A further solution is to map a planar 2D input space to a planar 2D space within the 
3D output space of the display, as depicted in figure 6. In this situation, the system maps the 
input space of a digitizing tablet 110 and the tilt/orientation of the tablet as sensed by a 
tilt/orientation sensor 112 to a corresponding planar space 114 in the display 116. The angle of 
the plane 1 14 is responsive to the sensor 112. If the display enclosure 130 has planar surfaces 
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(e.g., a cubic enclosure), the enclosure surface is used as the planar input device, as depicted 
in figure 7. It is also possible to use a transparent digitizer superimposed over an LCD display. 

[0026] Still another solution is to map a non-planar 2D input space to a 3D output space. In 
this solution, as depicted in figure 8, the system uses the display enclosure 140 as the input 
space (i.e., the enclosure is a transparent digitizing input surface). In this embodiment, the 
position 142 touched by the user or indicated by a pointing device, such as a stylus or surface 
fitting curved mouse, is mapped to a position in the display. This is a direct and compelling way 
to interact with these displays. 

[0027] In addition to direct manipulation using the hands, the solutions described herein also 
provide physical intermediaries between the hands and the input space as described below. 

[0028] Another solution when the user desires to interact directly with the enclosure surface, 
is to deform the surface 160 of a conventional deformable membrane surface that detects 
multiple pressure points and finger pinches 162, as depicted in figure 9. In this situation, the 
surface to display mapping discussed above is performed. 

[0029] Instead of using the hands directly on the enclosure surface, the system has a surface 
that detects and tracks a variety of input devices. A digital stylus 180, as shown in figure 10, 
where a point and an orientation can be input or a Rockin'Mouse shaped device 190, as shown 
in figure 1 1 (see U.S. Patent 6,1 15,028) also allowing a point and an orientation to be input are 
used. A surface fitting wireless mouse, such as a curved (concave) bottom mouse, can be used 
with a curved surface output configuration. This type of mouse can also be park-able using 
electrostatic, magnetic or some other sticky method of removably adhering the mouse to the 
display surface. Using a mouse has the advantage of buttons and form factors with which 
people are familiar. In this situation, the surface to display mapping discussed above is 
performed. 

[0030] The physical intermediaries also do not have to be on the enclosure itself as 
described below. 

[0031] In an embodiment input devices 200, such as buttons, keyboards, sliders, touch-pads, 
mice and space-ball type devices, etc., are mounted along the perimeter of the display (see 
figure 12). In this embodiment, the input devices such as buttons for up, down, forward, 
backward, left and right motions, allowing multiple degrees of freedom, are used to control the 
position of a cursor like such buttons control the position of a cursor in a 2D system. The input 
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devices 210, 212, 214 may need to be "repeated" (i.e., have more than one of each along the 
perimeter) to allow for simultaneous used by many users, or for use from any position the user 
may be standing/sitting at as shown in figure 13. Rather that having multiple input devices 
positioned around the display as depicted in figure 13, the mounting platform 220 that houses 
these devices could be made moveable (rotatable) around the display, as depicted in figure 14, 
so that users can easily bring the required device within reach by simply moving the platform. 
These devices typically communicate wirelessly by radio or infrared signals. The position of the 
movable device also provides information about the users position or viewpoint. 

[0032] The present invention is typically embodied in a system as depicted in figure 15 where 
physical interface elements 230, such as a rotary dome position encoder, infrared user position 
detectors, a keyboard, touch sensitive dome enclosure surface, mouse, beam pointer, beam 
pointer with thumbwheel, stylus and digitizer pad or stylus and stylus sensitive dome enclosure 
surface, stylus with pressure sensor, flock-of-birds, etc. are coupled to a computer 232, such as 
a server class machine. The computer 232 uses a graphical creation process, such as the 
animation package MAYA available from Alias|Wavefront, Inc., to create three-dimensional (3D) 
scene elements. This process, using position inputs from the input configurations as discussed 
herein, also creates the virtual interface elements, such as a virtual hand, a 3D point cursor, a 
3D volume cursor, a pointing beam, a bead, etc. The display output, including the scene and 
interface elements, is provided to a volumetric display apparatus configuration 234, such as one 
that will produce a 3D holographic display and discussed herein. 

[0033] The configurations that include a transparent digitizer or touch sensitive surface have 
a number of different shapes as depicted in figures 1 6A - 1 6D. In one embodiment a dome 
shaped enclosure 250 has a dome shaped digitizing tablet as depicted in figure 16A. In another 
embodiment the dome shaped enclosure 256 (see figure 16B) is used with a rectangular or 
cylindrical shaped digitizing tablet 258. In a further embodiment, as shown in figure 16C, a 
cylindrical or cubical enclosure 260 is used with cylindrical or cubical digitizer surface. In a 
different embodiment the enclosure 264 is dome shaped (or cubical or cylindrical) and the 
digitizing surface 266 is planar as depicted in figure 16D. 

[0034] A digitizer 280 (see figure 17), such as described in U.S. Patent 5,854,449 
incorporated by reference herein, determines a position of a stylus or pointer 282 relative to a 
surface 284, such as a transparent dome surface, having a checker board type closely spaced 
positional grid 286 thereon when seen from above. A processor 288 determines the coarse 
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position of the pointer relative to the grid by sampling the grid lines through a set of multiplexers 
290 and 292. An error correction system 294 generates and outputs a true position of the 
pointer 282 relative to the surface 284 to a computer system 232 (see figure 15). The pointer 
282 typically includes an electromagnetic transducer for inducing a signal in the positional grid 
286 and the processor 288 is coupled to the positional grid 286 for sensing the signal and 
generating the coarse position of the pointer 282. The transducers also allow the determination 
of a vector from grid signals that indicates in which direction the pointer 282 is pointing. Touch 
sensitive input surfaces operate in a similar fashion. 

[0035] The positional grid 286 can be applied to a surface of an enclosure, such as a dome 
shaped enclosure 310, as depicted in figures 18A and 18B. Figures 18A and 18B (an exploded 
view) show a section 312 of the dome surface including an inner substrate 314 and outer 
substrate 316 between which is sandwiched the grid 318. The substrates comprise transparent 
materials, such as glass or plastic. 

[0036] In using these input and output configurations the computer system 232 (see figure 
15) performs a number of operations as depicted in figure 19. The operations include obtaining 
330 the coordinate systems of the input device and the volumetric display. The range of the 
coordinate systems is also obtained so that out-of-space conditions can be determined. Next, 
the system samples 332 positional outputs of the input device, such as the digitizer, mouse, 
flock-of-birds, etc., to obtain the location of the users input. This information can also include 
information about where the user is pointing. This position (and orientation if desired) is 
mapped 334 into a 3D position within the volumetric display using the coordinate system (and 
the orientation vector, if needed). The cursor or other position indicating representation, such as 
a virtual hand, is drawn 336 at the mapped position with the volumetric display. The mapping 
may involve determining a position on the surface that is being touched by a digitizing stylus, 
projecting a ray into the enclosure from the touch position where the ray is oriented by the 
pointing vector of the input stylus and positioning the cursor at a variable or fixed position along 
the ray. Another mapping causes relative motion of a 3D input device such as a glove to be 
imparted to a cursor when a motion function is activated. Other mappings as discussed in the 
related applications are possible. 

[0037] The operations described with respect to figure 19, when a digitizing enclosure 
surface is the input configuration, allow the user to interact with a surface of a three-dimensional 
(3D) volumetric display and affect the 3D content of the display responsive to the interaction. 
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The interaction involves the user manipulating the stylus in a sensing region of the digitizing 
grid, the mapping of the stylus position to a 3D display position and the creation of a cursor at a 
3D display position. The cursor, in one of a number of different possibilities, is created at a 
distance offset from a tip of the stylus along a pointing vector of the stylus. The cursor can be 
used to perform typical functions such as selecting, painting, dragging/dropping, etc. 

[0038] The present invention has been described with respect to input configurations where 
commands are input through position sensing type devices, such as a mouse, a pointer, touch 
sensitive surface, etc. It is also possible to use other types of input configurations, such as non- 
spatial configurations. One non-spatial input space or configuration is a conventional voice or 
speech recognition system. In this configuration a voice command, such as "down" is 
recognized and the selected object or volume is moved accordingly. In this case down. The 
object is moved down in the display space at a constant slow rate until it reaches the bottom or 
until another command, such as "stop" is input and recognized. For user centric commands, 
such as "move closer", a user position sensing system inputs the user position, the position is 
used to determine the relative position of the active object with respect to the user or the vector 
pointing from user to the object. This vector is used to determine a direction for object 
movement. To move closer the object is moved along the vector toward the user by moving in a 
negative direction. Again the motion would continue until a blocking object is encountered or 
another command is recognized. 

[0039] Another non-spatial input configuration uses non-speech sounds, such as tones from 
a conventional multifrequency tone generator. Each multifrequency combination corresponds to 
a command and a conventional tone recognition system is used to convert the sounds to 
commands. 

[0040] The input space or configuration could also use conventional eye-tracking-head- 
tracking technologies alone or in combination with other input configurations. 

[0041] The many features and advantages of the invention are apparent from the detailed 
specification and, thus, it is intended by the appended claims to cover all such features and 
advantages of the invention that fall within the true spirit and scope of the invention. Further, 
since numerous modifications and changes will readily occur to those skilled in the art, it is not 
desired to limit the invention to the exact construction and operation illustrated and described, 
and accordingly all suitable modifications and equivalents may be resorted to, falling within the 
scope of the invention. 
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